Method for calibrating a vehicular camera system

ABSTRACT

A method of calibrating a vehicular multi-camera system includes equipping a vehicle with a plurality of cameras wherein each camera of the plurality of cameras captures image data, equipping the vehicle with an image processor, inputting image data from each of the plurality of cameras to the image processor, the image processor processing input image data in order to calibrate the vehicular multi-camera system, and wherein calibration of the vehicular multi-camera system is achieved independently of a model of the real world.

This application is a continuation of U.S. patent application Ser. No.12/604,432, filed Oct. 23, 2009, now U.S. Pat. No. 8,169,480, whichclaims the benefits of German Application No. 102008053047.6, filed Oct.24, 2008.

FIELD OF THE INVENTION

The present invention relates to a method for automatically calibratinga virtual camera and to a virtual camera apparatus which is set up forcarrying out the method according to the invention. In particular, thepresent invention relates to a virtual camera for producing a view ofthe surroundings of a motor vehicle from a bird's eye perspective on amotor vehicle.

BACKGROUND OF THE INVENTION

What is known as a virtual camera refers to sets comprising a realrecording camera and a habitually electronic image data processingdevice, which produce an output signal with a coded image or a codedimage sequence, wherein the perspective of the coded images does notmatch the perspective of the recording camera. On account of the loss ofinformation when a real three-dimensional object is mapped into atwo-dimensional image data model by a real recording camera, the virtualcamera is able to reproduce non-moving objects correctly particularlywhen they are approximately flat.

Virtual cameras have been proposed as driver assistance devices in motorvehicles. These are particularly what are known as top view systems oromnidirectional cameras. These typically comprise a plurality of realrecording cameras which are arranged in or on a vehicle and which areused to produce a chronological sequence of image data records. Theimage data in the image data records are subjected to differenttransformations in a typically electronic image data processing deviceand are mixed to form a chronological sequence of overall image data.This makes it possible to obtain, by way of example, a view of thesurroundings of the vehicle from a perspective above the vehicle roof.This chronological sequence of overall image data can be continuouslydisplayed to the driver of the motor vehicle on a display apparatus inorder to simplify shunt or parking manoeuvres.

It is evident that in a virtual camera with a plurality of realrecording cameras the quality of the overall image delivered isdistinctly dependent on the exact knowledge of the positions anddirections of the real recording cameras. The more accurately that thesedata are known, the easier that it is possible to determine thetransformation that determines the best possible image quality in thearea of adjacent recording areas. Against this background, there havebeen a series of proposals involving either the automatic determinationof the positions and directions of the recording cameras or thecorrection of errors in these variables in relation to initially storedvalues.

An omnidirectional camera with automatic calibration is revealed by theofficially published document DE 10 2007 043 905 A1. Said documentproposes identifying an object element in the image data for the purposeof the calibration. The recognition of mapped areas of the outer vehicleskin is recognized as particularly advantageous. The proposed approachis intended to allow compensation for, inter alia, situational changesin the real recording cameras on the vehicle on account of vibrations,ageing and thermal or mechanical stresses.

However, the known method is based on the processing of information froma model of the real world and particularly on the processing ofinformation regarding the shape of the outer vehicle skin.

SUMMARY OF THE INVENTION

Against this background, the present invention is based on the object ofproviding a method for the automatic calibrating of a virtual camerawhich manages independently of a model of the real world.

This object is achieved by the present invention by means of a methodhaving the features specified in claim 1.

Advantageous refinements and developments of the method according to theinvention are specified in the subclaims.

BRIEF DESCRIPTION OF THE DRAWINGS

A preferred manner of carrying out the method according to the inventionand a virtual camera apparatus which is set up to do so are describedbelow, reference being made to the appended drawings, in which:

FIG. 1 shows a schematic illustration of a typical arrangement of realrecording cameras on a motor vehicle and of the situation of the virtualperspective; and

FIG. 2 shows a schematic illustration of the functional units and datastreams in a preferred apparatus for carrying out a method according tothe invention; and

FIG. 3 shows a schematic synopsis of an algorithm for implementing amethod according to the invention.

DETAILED DESCRIPTION OF THE INVENTION

As FIG. 1 shows, an expedient apparatus for carrying out a preferredmethod according to the invention for a driver assistance system on amotor vehicle first of all comprises a plurality of real recordingcamera devices 1, 2, 3. A fourth recording camera device 4 is located inthe area which cannot be seen on the opposite side of the vehicle fromthe recording camera device 2. In one expedient implementation, the realrecording camera devices 1, 2, 3, 4 are in the form of a CCD or CMOSarray with a wide-angle or fisheye lens in a manner which is known perse and are arranged at different points on the motor vehicle 4 behindthe glazing and in suitable pockets on the exterior mirrors. In thefitting positions which are possible in practice, the real perspectivesP_(1 . . . 4) of the real recording camera devices 1, 2, 3, 4 arenaturally offset and/or tilted with respect to the virtual perspectiveP_(v), that is to say the viewing point of the virtual viewer. Thegreater that the difference between the real perspective P_(1 . . . 4)of a real recording camera device 1, 2, 3, 4 and the virtual perspectiveP_(v) turns out to be, the worse the quality of the image data with thetransformed perspective under realistic conditions. This will bediscussed later. Conversely, some fields of vision would not be visibleto a viewer with a perspective corresponding to the virtual perspectiveP_(v), because they are situated behind portions of the outer vehicleskin. In this respect, the selection of the camera positions is atrade-off between these advantages and disadvantages.

As further components, the apparatus described which is shown in FIG. 2comprises an image data processing device 5 and a display device 6. Inthis case, the display device 6 is typically arranged in the area of theinstrument panel or the central console. In a typical implementation,the image data processing device 5 is a digital signal processor or apowerful microprocessor for general applications with adequate equipmentin terms of main memory and non-volatile program memory.

In line with the schematic illustration of a preferred algorithmicimplementation of the method according to the invention which is shownin FIG. 3, the real recording camera devices 1, 2, 3, 4 respectivelydeliver a chronological sequence of N×4 raw subimage data itemsDR_(1 . . . N, 1 . . . 4) to the image data processing device 5 in theoperating period under consideration. Said image data processing devicecombines the data streams to form a chronological sequence of subimagedata records DR_(1 . . . N). Each subimage data record DR_(i) in thesequence contains the subimage data DR_(i, 1 . . . 4) from the realrecording camera devices 1, 2, 3, 4. In the image data processing device5, a respective transformation T_(1 . . . 4) is applied to the subimagedata DR_(i, 1 . . . 4) which a subimage data record DR_(i) contains whenit is present. These transformations T_(1 . . . 4) are respectivelydetermined such that the relevant image 7, 8, 9, 10 of a predeterminedplanar mapping area A_(j) of the real recording camera device j with theperspective P_(j) is transformed into the image from the virtual camerawith the perspective P_(v). For the case of an ideal recording cameradevice, this transformation would be linear and could be compiled from aplanar perspective extension of the image by means of rotation anddisplacement. In comparison with the aforementioned ideal recordingcamera device, however, the real recording camera devices 1, 2, 3, 4deliver maps with nonlinear distortions. The primary cause of thesenonlinear distortions is inadequacies in the real mapping lenses. Thisrelates particularly distinctly to the proposed lenses with a strongwide-angle or fisheye characteristic. From this point of view, positionsfor the real recording camera devices 1, 2, 3, 4 which are further awayfrom the road are naturally preferable. However, the nonlineardistortions can usually be attributed to a satisfactory measure forobservation of the environment using what are known as inverse nonlineartransformations, which are known per se. Consequently, a person skilledin the art will, according to the respective situation, select a chaincomprising a nonlinear transformation for the purpose of equalizationand an ideal perspective extension for the described transformationT_(1 . . . 4). The application of the transformations T_(1 . . . 4) tothe subimage data DR_(i, 1 . . . 4) results in transformed subimage dataDT_(i, 1 . . . 4). In the present case, it is assumed that thetransformations T_(1 . . . 4) transform the subimage dataDT_(i, 1 . . . 4) directly into the coordinate system of the overallimage 11. Accordingly, the overall image data DG_(i) can be produced bysimply combining the data from the transformed subimage dataDT_(i, 1 . . . 4).

Since, in the ideal case described, the positions and recordingdirections of the recording camera devices 1, 2, 3, 4 did not change,the information about the orientation and situation of the subimages inthe overall image would need to be set only once for the relevantvehicle geometry as a parameter for the transformations T_(1 . . . 4)prior to startup. In the course of such initial calibration, theparameters can be determined either by calculating or calibrating theapparatus on a test bench. Even with optimum initial calibration,ambiguities can arise in the area of the overlaps 12 in the subimages 7,8, 9, 10 on account of the described nonlinear distortions in the realrecording camera devices 1, 2, 3, 4. Through suitable selection of thetransformations and/or expedient stipulation of the subimage boundaries,however, it is possible to attain entirely satisfactory results inpractice. In this regard, an appropriately adjusted image dataprocessing device 5 produces as an output signal a sequence of overallimage data DG_(1 . . . N) which are displayed in their chronologicalorder on a display device 6 and give the driver an impression of theimmediate surroundings of the motor vehicle.

Effects of ageing, overloads, accidents and the like may result in theposition and/or orientation of the recording camera devices 1, 2, 3, 4being changed. If such a change is followed by the subimages continuingto be assembled to form an overall image in the originally stipulatedmanner, the result is poorer quality for the overall image. Tocounteract this drawback, the image data processing device 5 recurrentlyperforms calibration with the aim of optimization using a prescribedquality criterion Q for the overall image. In this case, the qualitycriterion Q is a scalar value which is dependent on the data from thesubimages and on the stored information relating to the positions anddirections of the real recording camera device 1, 2, 3, 4. Expediently,the quality criterion Q is stipulated such that it reflects the qualityof the overall image, as subjectively perceived by an average viewer. Inone advantageous refinement, the quality criterion Q will also relate tothe correlations of the subimages 7, 8, 9, 10 in the areas of overlap12. The quality criterion Q is optimized for a firmly prescribedsubimage data record DR_(i) by varying the parameters of thetransformations. The parameters varied to the optimum replace the valuesoriginally stored in the apparatus for the purpose of further operationof the apparatus.

This calibration is recurrently performed over time whenever a selectioncriterion C flags a subimage data record DR_(k) for this purpose. In thepresent case, the selection criterion C is defined such that the flaggedsubimage data record DR_(k) means that the calibration provides the bestpossible result. Intuitively, a good result will be assumed if theapplication of the quality criterion Q to the overall images 11 whichfollow the calibration provides the best possible result overall. Sincethe quality criterion Q in the present case relates only to anindividual image data record, the quality of a sequence naturallyrequires appropriate definition. To this end, it is possible to use thegenerally known statistical functions, such as the mean value.Expediently, the selection criterion C processes the subimage dataDR_(k, 1 . . . 4) anyway in order to assess the suitability of theflagged subimage data record DR_(k) for the calibration. This complieswith the insight that not all subimage data records DR_(i) are equallygood for the calibration in practice. By way of example, calibrationmust obviously remain undone in the event of a lack of contrast, underexposure, defocusing or motion blurring. Equally disadvantageous areimage data records with periodic structures, which can be recognized bymeans of frequency analysis of the subimage data DR_(k, 1 . . . 4), forexample. In addition, the subimage data DR_(k, 1 . . . 4) can beexamined to determine whether they contain maps of three-dimensionalobjects above the road level. Typical objects of this kind are highkerbstones, crash barriers, marker posts, for example. A subimage datarecord DR_(i) with maps of such objects in the area of the overlaps 12should not be used for the calibration.

In addition, the selection criterion C flags a subimage data recordDR_(k) for calibration only if there was a particular state of the motorvehicle at the time at which said subimage data record was recorded, andthe driving situation was within prescribed limits at this moment. Tothis end, the image data processing device 5 also derives, collects andassesses vehicle state variables and driving state variables fromdetection devices on the motor vehicle. In this context, preferredvehicle state variables are the operating period and mileage of thevehicle, the number of vehicle starts and the operating period and alsothe mileage since the vehicle was last started. By including thesevariables in the selection criterion C, it is particularly possible totake account of thermal changes, mechanical settling or ageing effectsand wear. Preferred driving state variables selected are the speed oftravel, the acceleration, the steering angle, the angle of inclinationand the loading of the vehicle. If available, it is also possible toinclude data about the tyre pressure and the setting of the suspension.When these data are included, it is possible to take account of thedynamic differences in the vehicle situation in relation to the roadsurface when deciding about calibration.

Further preferred variables for inclusion in the selection criterion Ccould be the GPS position of the vehicle, the exterior light conditionsand signals from proximity sensors for the near field of the vehicle. Byincluding such variables, it is possible to base the decision aboutcalibration on considerations concerning whether and to what extent thecurrent vehicle surroundings favour or impede calibration.

On the basis of a subimage data record DR_(k) flagged by the selectioncriterion C, the calibration can preferably be performed by calculatinga correlation between the subimage data DT_(i, 1 . . . 4) in a mannerwhich is known per se. In this case, the areas of overlap 12 areidentified and the situation and orientation of the image sections codedin the subimage data DT_(i, 1 . . . 4) relative to one another aredetermined. The quality criterion Q is optimized for the flaggedsubimage data record DR_(k) by varying the parameters of thetransformations T_(1 . . . 4).

In one refinement of the method described above, it is also possible toinclude the history of the calibrations performed in the past in theselection criterion. For example, this history could be used todetermine the time for the next calibration. It is also possible todetermine the parameters of the transformations not exclusively on thebasis of the result of the last calibration, but rather to performhistorical averaging. Yet another option is to anticipate adaptation ofthe parameters without calibration by extrapolation on the basis of thealready collected historical data from past calibrations.

1. A method of calibrating a vehicular multi-camera system, said methodcomprising: equipping a vehicle with a plurality of cameras wherein eachcamera of said plurality of cameras captures image data; equipping saidvehicle with an image processor; inputting image data from each of saidplurality of cameras to said image processor; said image processorprocessing input image data in order to calibrate said vehicularmulti-camera system; and wherein calibration of said vehicularmulti-camera system is achieved independently of a model of the realworld.